To achieve reliability in distributed storage systems, data has usually beenreplicated across different nodes. However the increasing volume of data to bestored has motivated the introduction of erasure codes, a storage efficientalternative to replication, particularly suited for archival in data centers,where old datasets (rarely accessed) can be erasure encoded, while replicas aremaintained only for the latest data. Many recent works consider the design ofnew storage-centric erasure codes for improved repairability. In contrast, thispaper addresses the migration from replication to encoding: traditionallyerasure coding is an atomic operation in that a single node with the wholeobject encodes and uploads all the encoded pieces. Although large datasets canbe concurrently archived by distributing individual object encodings amongdifferent nodes, the network and computing capacity of individual nodesconstrain the archival process due to such atomicity. We propose a new pipelined coding strategy that distributes the network andcomputing load of single-object encodings among different nodes, which alsospeeds up multiple object archival. We further present RapidRAID codes, anexplicit family of pipelined erasure codes which provides fast archival withoutcompromising either data reliability or storage overheads. Finally, we providea real implementation of RapidRAID codes and benchmark its performance usingboth a cluster of 50 nodes and a set of Amazon EC2 instances. Experiments showthat RapidRAID codes reduce a single object's coding time by up to 90%, whilewhen multiple objects are encoded concurrently, the reduction is up to 20%.
展开▼
